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Description 
TECHNICAL FIELD 



9 [0001] This Invention relates to data packetization and, In panicuiar, to packetization of fine granularity st atable 
(FGS) and progressive fine granularity scalable (PFQS) bitstreams. 

BACKGROUND 

10 [0002] Due to increasing popularity of multimedia streaming over networks, such as the Internet, streaming perform- 
ance is an important issue. Typically, there are two factors that affect streaming performance. The firrt is the fact that 
available network bandwidth may vary over time depending on the amount of network traffic. The second is the fact 
that current delivery processes are not error free, due to the best-effort nature of the currant Internet. This results In 
data packets being occasionally lost during transmission. Due to the nature of common video encoding methods, It Is 

w common for d^ In one data packet to depend on data that is contained in a previous data packet, accordingly, per- 
formance of streaming video suffers when data packets are not received. 

[0003] Various video encoding methods can be implemented to address the first issue, that of varying available 
bandwidth. Such methods indude a fine granularity scalable (FGS) encoding scheme, a frame-based progressive fine 
granularity scalable (PFGS) encoding scheme, and a macroblock-based PFGS encoding scheme, each of which is an 
^0 Improvement upon the previous In terms of adapting to available bandwidth variance. Although FGS/PFGS encoding 
methods can effectively improve performance of streaming video over networks with varying bandwidth, they do not 
address performanoe decreases caused by lost data packets. 

[0004] Independent development efforts have also focused on ways of improving the delivery of data packets over 
a network. This work has shown that performattce improvement has an inverse relationship to the degree of inter- 
» packet dependency. That is, streams of data packets that do not depend on one another result In greater performance 

than streams of data packets that depend heavily on one another. 

[0005] Accordn^, there is a need to establish a method for packetlzlng FGS/PFGS encoded video bitstreams such 
that dependency among the resulting data packets Is minimized. 

00 SUMMARY 

[0006] Packetization of FGS/PFGS video bitstreams to minimize Inter-packet dependency is described A video en- 
coding system encodes a video stream according to an FGS/PFGS coding scheme, resulting In encoded bitstream 
segments. A set of the encoded bitstream segments is then selected based on each bkstream segment's total contri- 
35 button to distortion reduction and based on an available bandwidth. The selected bitstream segments are then pack- 
etized into data packets such that inter-packet dependency is minimized. 

BRIEF DESCRIPTION OF THE DRAWINGS 

90 [0007] The same numbers are used throughout the drawings to reference like features and components. 

[0008] Rgure 1 illustrates an exemplary video ertcoding system In which packetization of FGS/PFGS video streams 
may be implemented. 

[0009] Figure 2 ilustrates an exemplary architecture of a fine granularity scal^le (FGS) coding scheme. 

[0010] Figure 3 illustrates an exemplary architecture of a frame-based progressive fins granularity scalable fPFGSI 
90 coding scheme wiOi two references. ' 

[001 1] Rgure 4 IBustrates an exemplary architecture of a macroblock-based PFGS coding scheme. 

[0012] Rgure 5 ilustrates dependencies between bit planes generated by FGS or PFGS coding schemes. 

[0013] Figure 8 ilustrates packetization of two frames according to a btweline packetization strategy. 

[001^ Figure 7 ilustrates an error propagation model for estimating the drifting suppression cwsociated with PFGS 
so bitstream segments. 

[0015] Figure 8 ilustrates packetization of two frames according to a binary-tree packetization strategy 
[0018] Figure 9 illustrates packetization of several frames according to a macroblock-based packetization strategy 
[0017] Figure 10 illustrates an exemplary method for encoding and packetlzlng a video bitstream. 

[Q018] Rgure 11 Ilustrates an exemplary method for gathering data associated with an encoded video bitstream to 
55 support selection of a set of bitstream segments to be packetized. 

[0019] Figure 1 2 Ilustrates an exemplary method tor selecting a set of bitstream se^nents to be packetized. 
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DETAILED DESCRIPTION 

® synergistically integrated. ®<teptatk>n and optirnal delivery can be 

Exe mplary Syetem AidiHecfaire and Methods 

Stream are then transmitted to client device 104 via network 106 containing the encoded video 

In an exemplary implementation. FGS/PFGS aneodar iia irr.nin..i , i i " " res or PFGS encoding scheme. 

l^oaSraa^f^^J^ If® other applications 1 30 are stored In memory 126 and executed on process 

streaming Video Encoding Methods 

FGS encoded frame may bo predicted from the base laver of a nr«ui«i .f» ’ a ^ quality enhancement layer. An 

Is described in further detail below, with raferance to Figure 2 ^ An exemplary FGS encoding schamo 

described In further detail below, with refera^ to FiguIUT’ ^ ^ frame-based PFGS encoding scheme is 
£0028| A macroblock-based PFGS encoding scheme improves upon the described frnmn hnmri oer-e ^ 
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Fine Granularity Sealable (FQS) Coding Scheme 

[00291 Figure 2 illustrates an exemplary architecture of an FGS coding scheme. FGS is a coding scheme that results 
in a bitstream that can be arbitrarily truncated based on available bandwidth, such that each truncated version of the 

9 bitstream Is still fully decodable. As Illustrated In Figure 2, an FQS encoder generates two bitstreams* the base layer 

(represented by blacks 202(1 ). 202(2), 202(3), 202(4), ...) and the enhancement layers (represented by blocks 204(1 ) 
204(2), 204(3), 204(4) 206(1 ), 206(2), 206(3), 206(4) 208(1 ), 208(2), 208(3), 208(4), ...). 

[0030] The base layer 202 is coded by traditional motion compensated discrete cosine transform (DCT) common 
in other layered coding techniques. The base layer 202 is typically of low quality and is very thin to ensure that it will 

10 fit in even very small bandwidths. The residue between the original DCT coefficients and the dequantized base layer 
DCT coefficients forms the enhancement bitstream (204, 206. 208, ...). which is coded using bit-plane coding technol- 
ogy to yield an embedded bitstream. In an FGS encoded bitstream, the base layer and an enhancement layers in a 
predictsd frame are coded from a reconstructed version of the lowest-quality base layer of a reference frame. 

10 Fnme-Bmsmd PFOS Coding Scheme 

[0031] Rgure 3 Illustrates an exemplary architecture of a frame-based progressive fine granularity scalable (PFGS) 
coding scheme with two references. PFGS Is a coding scheme that improves upon FGS by predicting a frame's en- 
hancement iayerts) from a reference frame's enhancement layer, while continuing to predict the frame's base layer 
eo from the reference frame's base layer. For example, as shown In Figure 3, the base layer of the second frame 302(2) 

is predicted from base layer 302(1 ) of the first frame, and enhancement layers 304(2), 306(2), 308(2), . of the second 
frame are predicted from enhancement layer 308(1 ) of the first frame. Sfrtillarly, the base layer of the third frame 302 
(3) is precficted from base layer 302(2) of the second frame, and enhancement layers 304(3), 306(3), 308(3) of the 

third frame are predicted from enhancement layer 306(2) of the second frame. Because the quality of a frame is higher 
25 at an enhancement layer than it is at the base layer, the frame-based PFGS coding scheme provides more accurate 
motion prediction than the FGS scheme, and thus improves the coding efficiency. 

[0032] However, when transmitting a PFGS encoded bitstream, a high-quality enhancement layer reference for pre- 
dicting future frames may be partially or completely lost due to truncation caused by a decrease In available bandwidth 
In such a scenario, the decoder would have to use a corrupted high-quality reference or a low-quality base layer 
50 reference to predict the future frames. Because the encoder used a high-quality reference to predict the future frames 

a drifting error will likely be Introduced. To reduce the drifting errors, the PFGS coding scheme also Incorporates a 
bask; drifting control method of alternatively reconstructing a high-quality reference from a previous low-quality refer- 
ence In an event that a previous hl£ffi-quality reference Is not available. For exarr^le, as Illustrated In Figure 3, If 
enhancement layer 308(2) is corrupted or lost during transmission, a new high-quality refererxw 306(2) is alternatively 
55 reconstructed from base layer 302(1) and enhancement layer 308(1) of frame 1 . 

Macroblock-Baaed PFOS Coding Scheme 

[00331 Macroblock-based progressive fine granularity scalable (PFGS) coding Is a coding scheme that improves 
40 upon frame-based PFGS by utilizing three coding modes and a decision-making mechanism for selecting a coding 
mode to be appOed for each enhancement layer macroblock. The three modes are; iow-prediction-low-reconstruction 
(LPLR), high-prediction-low-teconstructlon (HPLH), and high-prediction-hlgh-raconstiiiction (HPHR). By selectively 
choosing an appropriate mode for each macroblock to be coded, drifting errors are reduced at low bit rates and coding 
effldoncy Is improved at moderate and high bit rates, when compared to the FGS and frame-based PFGS codina 
^ schemes. 

[0034] Figure 4 illustrates the LPLR, HPLR, and HPHR modes of a macroblock-based PFGS coding schema Mao- 
roblock 404 is coded using the LPLR mode, macrobiock 406 is coded using the HPLR mode, and macroblock 408 is 
coded using the HPHR mode. In Figure 4, prediction is represented by solid arrows and reconstruction is represented 
by hollow arrows. 

50 [0038] According to the LPLR mode of a macroblock-based PFGS coding scheme, enhancement layer macroblocks 

are both predicted and reconstructed from a previous low quality reference. This is illustrated by the relationshto be- 
tween macroblock 402 (the reference) and macroblock 404 (the macroblock being coded). Base layer 404(1) and 
enhancement layers 404(2), 404(3), 404(4) are all predicted from base layer 402(1), which Is a low-quality refer- 

ence. Furthermore, enhancement layer 404(3) is also reconstructed for use as a reference to future macroblocks from 
55 base layer 402(1 ) (incficated by the hollow arrow). 

[0036] According to the HPLR mode of a macroblock-based PFGS coding scheme, enhancement layer macrobloeks 
are predicted from a previous high-quality reference, but are reconstructed from a previous low-quality reference This 
is ilkrstrated by the relationship between macroblock 404 (the reference) and macroblock 406 (the macroblock toing 
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~ded). Base layer 406(1) is predicted from base layer 404/1) which is r 
layers 408(2). 406(3). 408(4) are predicted frorTent^KinT^^ 

are both predicted and rwretnSS 
between rnacrobIrxK 408 (the refere^^d „S^ 

HPLR modes, base layer 408(1) is predicted As in both the UPLR and 

are predicted from enhancement layer 406(3) whk^ is a hS[-^iIi 

hollow arrow). macroblocks from enhancement layer 406(3) (indicated by the 

Baseline Packetization StrateoY 

address the negative impacts of SictaSrSlSJ^tton?^^^ "JJ encoding schemes that 

mesTO for minimizing the effect of errors caused by packet loss duSJ SSS? ’ ** « 

[0039] FIguro 5 lllustratas th© dopondencias batwoan FGS/PFGS a mi 

trated in Figure 5. a bitstream segn!^ is decodable onh/ it auT#^ dependencies illus- 

to the same macrobiock but In the lower bit-planes) are raceh/ad ^ segments belonging 

S04 is decodable only If bltstre^ ^ Ii aS^s^S^ •’tstrearn 

o^ blt^eams M2. 504. and 506 are received and suocessfulV diSS?*^' 

In significant packet^d?^SJdJJ!^^flJi^ strategy, which results 

associated with Frame 1 . Similarly, packets P. pIr. P P p p p ' o o®’ data 

“ »n!sis «"SiIp 

Ofoy ». «*.„«« oo pocto. p“paSirS. p!TT? P, P K»t, «».. U»oi» 

•oocddully. TI» fo« PMM al«, oool«rtn«i ^ r«»fv«l 

on ttie implementation, of the decoder. The contaminatfon of oaekat P ^ partially decodable. depentfng 

Lategy is ^iSdjrpjJTby ac*»rdlng to the described baseline packetization 

representing the performance of an encoded bitstream will nw bTdwetoSlTmS^ T“' ^ tor 

establishing two exemplary packetization stratealas to radueo .. . ^ subsequently used as a basis for 

they are generated. between packets by aligning p^^ 

FQB/P FQS BHstream Performance Metric 

technology. For each 8x8 OCTbk^k. t^iM i^oSte^ ve ztotHTdellld kiT** <^<^9 

is def^ M an array of 64 bits, one from each absolute value of thtocr^JSiJJirt ^ ^ 

For each bit-plane of each macroblock. (RUN. endHrf-bllplane fEOPM^!^^!! ! P«»ftlon. 
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minimum manipulation unit will be such a bitstream segment. Because of the bit-plane coding used for producing the 
enhancement layer bitstreams, the dependency among bitstream segments associated with the same macroblock is 
very strong; whereas, there is no dependency among bitstream segments associated with different macroblocks. (As 
described above, the dependencies between FGS/PFGS bit-planes is illustrated in Rgure 5.) 

[0046] The mathematical probability that a particular bitstream segmem Is decodable can be detennined based on 
this information. To successfully decode a bitstream segment S, In the P* bit plane, S, must be received without error, 
and Sf .1 (i.e., the bitstream segment that belongs to the same macro block but In the lower (/-I )•* bit plane) must nigni 
be decodable. Furthermore, in order for bitstream segment to be decodable, must be received and S/^ must 
also be decodable, and so on. In mathematical terms, this can be expressed as: 



p(Si decodable) 

=p(Si received, Si.\ decodable) 

= p(St received. Si. i received, 5^.2 received, . . . , S’© received) ( 1 ) 

[0047] For example, the decodable probabilities for three bitstream segments. So, Sf, and Sg, all belonging to the 
same macroblock but in the 0***, I**, and 2"^ bit planes, respectively, may be expressed as follows: 



PiSp deeodE4>le) - p(Sp received) 



p(Si decodable) » p(S\ received, 5o decodable) 
= p(Si received, 5o received) 



/)(52 decodable) —p(52 received, .S| decodable) 

^piSz received, Si received, 5b received) 



[0046] To simplify the above decodable probabflities, the joint probability p(Sg received, Sy received) is analyzed. If 
■«> the bitstream segments S, and Sy are packetized into the same packet, packet k, then the joint probability may be 
expressed as: 



p{Sx received, Sy received) 

= p(Sjc received) « p(Sy received) = /^(packet k received) 



so [0049] If, however, bitstream segments S, and Sy are packetized Into different packets, packet J and packet k, then 
the joint probability may be expressed as: 
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/>(& received, ^ received) 

~ p{Ss received) X p(Sf received) 

= />(packet J received)x /^(packet k received) 



10 



15 



tn« packet occnialna Die aricaMof bitttream «nmar«tai r«KrcTV 

proPablW. o( s,. ,M ,t, „„ The, th. aaeodap. 
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p{Si decodable) = p(St received) x 1 1 /?(packet m received) 



p(5’z decodable) =• p(S 2 received) x /?(packet m received) 
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( 2 ) 



wharein the term /, ^ denotes the bitstream seonent v^h the ith framai «i ki* i ^ 

AO<f./. !i represents the distortion reduction thatteLhle^ /IT /m ^ ^ nfiaerobiock. 

respectiveiy. the pactot ios. prob^iuj oJIre^o^rtSi ««• 

^ m denote, the bitstre^n segrli^r^Z^er:::^:^^^^^ « 4 0 - « 

sot. I, represents afi enhancement iayer bitstream segn^ th^ wiil bTh^n^^Sell! btetream segments 

corresponds to a bandwidth adjusting Interval). ThelSI^JlSni rate Irf^ah^llw^^ “"®"* 
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£00821 Equation 2 provides a generaJ performance metric that consider, the dependency among btt planes in the 
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enhancement layer bitstream, the dependency between the base layer and the enhancement layer bitstreams, and 
also the error protection on the enhancement layer packets. The influence of error protection is reflected through pe 
(•). If unequal error protection (UEP) to adopted, p,'s wil be different for different enhancement iay'er packets. On the 
other hand. If equal error protection (EEP) or no enror protection is adopted, p^s will be the same for all enhancement 
* layer packets. Since the enhancement layer bitstream depends on the base layer bitstream, which is usually very thin, 
it is assumed that the base layer bitstream is transmitted correctly. Furthermore, for the sake of sinvHcity, no error 
protection is applied on the enhancement layer packets. 

[OOS3] Accordingly, the performance metric defined above indicates two factors that should be considered when 
selecting a packetization system. Firstly, the bitstream segment set, /, that provides maximum dstortion reduction for 
10 the available bandwidth should be selected. (Because the selection is based on the availabie bandwidth, the selection 

of bitstream segment set, /, can be explained In tenns of rate allocation.) Secondly, a packetization strategy should be 
selected that minimizes the impact of packet loss (by minimizing the dependency between packets). 

Rate Allocation 

15 

[0054] For each encoded bitstream segment. If the rate and contribution to distortion reduction are known, the se- 
lection of bitstream segment set, /, can be converted into a standard Langrangian minimization problem, and a preferred 
solution can be found by applying an equal-slope argument. For each FQS or PFGS enhancement layer bitstream 
segment, the corresponding rate is explicit while its contrtoution to distortion reduction is not so obvious, especially 
so for PFGS bitstreams. 

[0058] For an FGS bitstream, the contribution of a certain enhancement bitstream segment consists only of the 
resultant distortion reductton for the current frame, and can be calculated precisely (based on the unitary property of 
the discrete cosine transform (DOT)) using the number of 1 's in the residual bit plane In the DCT domain: 

= n X (4j 

where n Is the number of 1 ’s of the current MB in the P* bit-plane; /= 0,1 ,2,...; L -1 denotes the index of the bit-plane 

(/= 0 denotes the most significant bit-plane and /- L -1 denotes the least significant bit plane); and L is the maximum 
so number of bit planes in the frame. 

[0056] Because a PFGS enhancement layer may be used as a high quality reference fora future frame. It can 
quality loss of the current frame (as is the case with an FGS enhancement layer) and can also cause drifting In the 
subsequent frames. Therefore, the contribution to distortion reduction of a particular PFGS bitstream segment is based 
on two parts: the distortion reduction for the current frame (as for an FQS bitstream segment and drifting suppression 
35 for the subsequent frames. The contribution to distortion reduction of a PFGS bitstream sejynent may be expressed as: 

= ^OolfJM&D^f.l.l) ( 5 ) 

^ where the terms ADo/^,1,1) and denote, respectively, the distortion reduction and drifting suppression of bit- 

stream segment (f,/,4. 

[0087] Directly calculating the drifting suppression (HD^W)) is unrealistic in a practical system due to its extremely 
high computational complexity, and therefore must be estknated. Such an estfrnate will be detemiined based on an 
error propagation pattern associated with a PFGS coding scheme. As described above, the macrobloek based PFGS 
coding scheme resulU in better performance than the frame-based PFGS coding scheme. Accordingly, the estimated 
drifting suppression is determined based on the macroblock-based PFGS coding scheme. 

[0058] As descrftred above with reference to Figure 3 and 4, according to the PFGS coding scheme, a high-quality 
reference at an enhancement layer is used for prediction. As a result, more accurate prediction Is obtained and batter 
coding efficiency is achieved as compared to an FGS coding scheme. On the other hcuid, drifting error is introduced 
if the high-quality reference bitstream is not received by the decoder (I.e., the encoder and decoder lose synchroniza- 
tion). To suppress the drifting error, the frame-based PFGS coding scheme restricts the maximal error propagation by 
periodicaily referencing a low-quaiity reference reconstructed from the base layer bitstream in both the enc od er and 
the decoder, in this way, the encoder and the decoder are wen synchronized and the drifting error propagation is well 
controBsd. However, the macroblocfc-based PFGS coding scheme does not irwiude a mechanism to guarantee the 
so periodical rraynchronization between the encoder and the decoder, and as a result, drifting errore may continue to 
propagate. The drifting error propagation Is further corrplicated by the fact that the macroblock-based PFGS coding 
scheme implements three coding modes, namely, LPLR, HPLR, and HPHR. Drifting errore are not propagated with 
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rurttiof to future frames. 9 «l <y *®y«r reference, the prediction error may be propagated 

[00S9] To estimate the drifting suppression for a oartb^nkir hitn. 

the current frame, the selection of coding mode LPLfl HPHR or HPLRb^"*’ “•«*• for each pixel in 

«hl(<) respectively represent the percentage of the HPHR and HPLR mo«JIfri1h**iT* ** ^ and 

in the P» frame that is referenced by the (A-1 )ih frame the omhaMiHu »e P» frame. Therefbre, for any pbcel 

and the probability that it is referenced In HPLR mo<te to R^fri) ** 

«». «. s:,zrcu° 

the drifting caused to the (fri )«• frame is approximatety eu’/ftx « ‘*1®*®'*^" '®<foo*lon is not received, 

%(fr1) , respectively. Additionallv e.fA. 1 ) wriil ha iiirthal.^ ^ and ^(i) x R/«,(A-1). denoted as e.^M) and 

(fj,0 can be represented as: ^ onhancement layer reference bltstrecun segment 

“ood for drifting estimation. 

100621 it then follows from equations (5) and (6) that the total contribution of btetream segment ifM te: 



A^./,0=.ADo(/. 
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ot a Md bIMraant ••gnnnta MMn Ib^ooratMW “gmoM ai» oMtfnod. th« •Mellon 

m..«mi.„ p«.im, •„. . ‘"• 

distortion reduction for the available bandwidth sl^uid be selected **'®* maximum 

selected that minimizes the impact of ® f«^«»on -rategy should be 

discussion Of selecting a packetization strategy^lotS ®*** '* '* above, and a 

discussion wiH deserftres two packotba^j!!^e^eBTt^^bl^ Performance. The foltowSi^ 

zatlon strategy (deserbed above with reference to Figure 6?^ desi^^d^l^ a^atlves to the baseline packet!- 
is simple and straightforward, but introduces heavy dSranrton^TJl^r^^S!^”* baseline packetization strategy 
a binary tree packetization strategy that greatly^o^!L tire dwenrtoSJ^fSli discuseion will deecrtoe 

packetization strategy that eiim^es dJpeS!ncy ISSSng paSte^^ ® ® P“*e**fn<tependence 
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reference to Figure 6. Bitstream segments of FGS enhancement bit planes typically tend to increase in size approxi- 
mately exponentially. Accordingly, the enhancement layers of a frame can be naturally represented by a binary tree 
wherein each node reprments a segment of the bitstream and the edges represent dependencies. A well aligned 
binary tree structure is constructed by aligning a left child to the left of its parent arxt aligning a right child to the riaht 
of Its parent in a top down fashion. ^ 

[0067] As illustrated in Rgure 8, the dependency among packets is significantly reduced. For example, if packet P, 
is lost, packets Ps, P4, P,, and P® are also impacted, but packets P2, P7, Pa, Pg, and P^q remain fully decodc6>le A 
downside to the binary tree packetization strategy is that the packet size varies from node to node. 

[poeq Each node In the binary tree is associated with a (rate, distortion) tuple. According to the available bit rate, 
the binary tree is pruned using a rate-distortion (R-O) criterion to select the nodes that result In the largest distortion 
reduction for the given bit rate. For example, for the frames illustrated in Rgure 8, some bitstream segments at the 4» 
bit-plane are left out. In one implementation, a prefened transport packet length (e.g., 6 kOo-bits) is defined to allow a 
parent node and two child nodes to be grouped together into one transport packet. 

Packet-independence Packettzatton Strategy 

[0069] Figure 9 illustrates packetization of multiple macroblocke according to a packet-independence packetization 
strategy that eliminates dependency between packets. As discussed above, because of the bit-plane coding used In 
both FGS and PFGS coding schemas, there is strong dependency among enhancement layer bitstreams segments 
of the same macroblock, but no dependency among bitstream segments of different macroblocks. Accordingly, the 
packet-independence packetization strategy petoketizes bitstream se^nents from all of the bit-planes of the same 
mactobloek Into the same packet. Therefore, the performance metric defined by Equation (2) can be reduced to: 



= 5; AZ)(/./, 0 X (1 - pe(/.U)} 



( 8 ) 



where the rate constraint remains unchanged. 

[0070] As described above, the AO(W) term represents the distortion reduction that is achieved If is successfully 

decoded. Using a macrobtock as the smallest unit, an R-O based selection strategy Is appUed at the macroblock level. 
Each l^er of a macrobtock is represented by a (rate, distortion) tuple, and for a given target bit rate, the equal-slope 
argument In a Langranglan minimization problem Is used to determine which macroblocks, and which layers of each 
macroblock should be seiected such that the Lagranglan ob|ectlve function IMJI Is mlnimtzed. In other words, regard- 
ess which layers and which macroblocks the bitstream segments are from, only those bitstream se^ents that result 
in the most significant distortion reduction are selected while observing the bandwidth constraint As illustrated In Figure 
9, different nurrtoers of layers may be selected for each of the different macroblocks. For example, macroblock 10 
includes only three layers, while mecroblocks 0, 3, 6, and 7 each include five layers. 

[0071] After bitstream selection, packets are generated in such a way that all of the seiected bit-planes of the same 
mawblock ^ packetized into the same packet so as to eliminate packets dependencies. When a pre-determined 
packet length is reached, a packet Is formed, and any remaining selected bitstreams from other macrobtocks are 
packetized in new p^kets. Bitstream segments from multiple macrobtocks may be packetized in the same p m-k et 
provided that all selected bitstream segments from a given macroblock are packetized in the same packet 
[0072] Aa illuatrated by the arrows in Rgure 9. raster-scan ordering is employed to facnitate packing the bitstream 
segments into the same packet The raster-scan ordering results in the bitstream segments being packed in order 
according to their R-D contribution from the most important bitstream segment to the toast important bitstream setynent. 

Methods ter Packetizetton of FCB/PFQ8 VIdee Bttetreame 

video bitstream packetization as performed by a video encoding computer system 102 may be 
described in the general context of computer-executable instructions, such as application modules, being executed by 
a computer. Generally, application modules include routines, programs, objects, components, data structures, etc that 
perform particular tasks or Implement particular abstract data types. An FGS/PFGS video bitstream packetization sys- 
tem may be implemented using any number of progranwning techniques and may be implemented in local computing 
environments or In distributed computing environments where tasks are performed by remote processing devices that 
are linked through various communicattons networks based on any number of communicatton protocols In such a 
distributed computing environmont. application modules may be located in both local and remote computer storaoe 
media including niemory storage device. ^ 
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^ 10 Will bo described with reference to components Illustrated in Figure 1 . ^ 

OC^ At Wo<* 1^, a video stream is encoded according to an FGS/PFGS coding scheme ForexamDia R3S/ 

PFGS ^der 1 1 6 of video encoding computer system 1 02 receivee a video streamto be encoded and^lies a 

2 ® beiow with reference to Figure 11. 

At block 1(^, bitstream eelector lie determines a set of bitstream segments for packetization in the de- 
screed Implementation, the set of bitstream segments is selected based on each bitstream's oosltion fframa bitnUino 

. “ ‘““•'"on rMoetloo. An ««ovliinSl>?SrLSSjri; 

se^bitstream segments is described in further cteteil below with reference to Figure 12 ® 

[007q ^block 1 008, packet generator 120 packetizes the selected sat of bitstream segments accordlno to a r,«^ 

Inter-packet dependency. One knplementation utiRzes the described binary tree pack 

^izatlonstotegy. whileanaKemate implementation utilizes the described packetMndependew^llzatonatirt^ 

In one implementation, raster-scan ordering is employed such that the selected bitstream seonents are ontored^^ 

-dooBon, Aon, in. n«« 

S. ~ ‘“.o- 

[0^] At block 1102, FGS/PFGS encoder 116 gathers position data (Including which frame which bkniarM nnri 

^ “*^ “ “» “«*' ‘"“‘i »» 

[0M1J At block 1104. FGS^FGS encoder lie determines a percentage of HPHR modes and HPLB mod.. tor--..h 
u'aTn;;,^^ ‘teheme, both of 

Son Tor‘’SrrSd^nr°® contribution to distortion 

" ••*«•» <««nB «wi~ion. For mty pnviou taiij(^n 

S! !?k^ ®" exemplary method ter selecting a set of bitstream segments to be pacfcetizad 

"* <,»oci«M wm nm.,k loe ,« . 

iis: irjsrsrs zrjz:. « 

(00681 '^*>'oc*‘1204.bit8tream8elector118load8thedatastoradinthe8idelnfomiationflle mil.lneiurt«.o«««i*irA 

‘’'***'^ ^^8 examines the rates and contributions to distortl^eduction for different 
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Condmion 



[0091] Although the systems and methods have been descrt>ed in language specific to structural features and/or 
methodological steps, it is to be understood that the invention defined in the appended claims is not necessarily limited 
to the specific features or steps described. Rather, the specific features and steps are disclosed as preferred fomw of 
impiementing the ctaimed invention. 



Claims 

1. A method comprising: 

generating a plurality of encoded bitetream segments by applying a cocfing scheme to a video stream; 
selecting multiple bitstream segments from the plurality of encoded bitstream segments; and 
packetizing the multiple bitstream segments Into data packets such that dependency between the data packets 
is minimized. 

а. The method as recited in claim 1 wherein the coding scheme comprises a fine granularity scalable coding scheme. 

3. The method as recited in claim 1 wherein the coding scheme comprises a progressive fine granularity « rfe la b le 
coding scheme. 

4. The method as recited In claim 3 wherein the progressive fine granularity scalable coding scheme is frame-based. 

5. The method as recited In clafrn 3 wherein the progressive fine granularity scalable coding scheme is macroblock 
based. 

б. The method as recited In claim 1 wherein the selecting compiisea: 

selecting multiple bitstream segments from the plurality of encoded bitstream segments based on an available 
bandwidth. 

7. The method as recited In claim 1 wherein the selecting comprises; 

selecting multiple bitstream segments based on contributions to distortion reduction associated with ktdividual 
ones of the encoded bitstream segments. 

8. The method as recited In claim 1 wherein the selecting comprises: 

determining a bttrate associated with individual ones of the plurality of encoded bitstream segments; 
estimating a contribution to distortion reduction associated with individual ones of the plurality of encoded 
bitstream segments; 

determining an available bandwidth; 

determining a number of frames to be transmitted; and 

selecting the multiple bitstream segments such thatfor each bitstream segment that is selacted, the associated 
bitrate is supported by the available bandwidth and the associated contribution to dstortlon reduction ia neater 
than or equal to a contribution to distortion reduction associated with at least one of the pturallty of bitstream 
segments that is not selected. 

9. The method as recited In claim 1 wherein the packetizing comprises: 

generating multiple data packets such that a first data packet contains bitstream segments 

multiple bit-planes of a first macro block, and such that no other one of the multiple data packets contains 

bitstream segments associated with the first macro block. 

10. The method as recited in claim 9 wherein the first data packet also contains bitstream segments with 

multiple bit^lanes of a second macro block. 




EP 1 458 196 A2 






5 



11. The method as recited in claim 1 wherein the packetizing comprises: 



representing muMpie enhancement layers associated with a frame 
tree represents a bitstream segment, and wherein each child node 
of a parent node; and 



as a binary tree wherein each node of the 
of the binary h-ee is aligned with an edge 



generating multiple data packets such that each data packet that contains 
node associated with the child node. 



a child node also contains a parent 
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12. A method comprising; 

“•***- »«•«'»» p—'o- «< •"»««< 

p2c^“ ^ ^ ** 

14. A method comprieing: 

identifying a plurality of encoded video bitstream segments to be packetized- 

b««», Mgm.™ .ccoMiig to . M„«, ^ 

packetizing the bitstream segments Into data packets according to the blnaiv tmn ei fha# m * . 

comprisesaparerrt node andachild node of ttwbinanf^. ^ ^^^^ 

15. The method as recited In claim 14 further comprising: 
poMJjgw b««, 

16. A method comprising: 

segments to be packetized; and 

17. The method as recited In claim 1 6 wherein the packetizing further comprises: 

P“*« “0^ “ • 

^ 16. A system comprising; 

an encoder configured to generate encoded bitstream se^nents; 
a bitstream selector configured to select a set of the encoded bit^eam seonents for 

®^®*®*®®'’*'®*°^“'’”9“^^®P®‘*«**2ethosetoftheencodedbit8tieam8eament8 Mcoidinotoa ^ 
etization strategy that minimizes dependency between packets. P®ck- 
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W. Tl» M ™cl«l io cteta 1., 0» ,«,» to 
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The system as recited In claim 1B, 
ularity scalable encoder. 



wherein the encoder Is implemented as a frame-based progressive fine gran- 



13 




22. The system as recited in claim 1 B, wherein the encoder is implemented as a macrobiock-based progressive fine 
granularity scalable encoder. 

23. The system as recited in claim 18, wherein the encoder is further configured to gather dam describing a position, 
a required bitrate, and a contribution to distortion reduction associated with an encoded bitstream segment. 

24. The system as recited in claim 1 8, wherein the packet generator Is further configured to packetize the set of the 
encoded bitstream segments accordktg to a binary tree packetization strategy that minimizes dependency between 
pcicicfits » 

25. The system as recited in clabn 18, wherein the packet generator is further configured to packetize the set of the 
encoded bitstream segments according to a packet-independence packetization strategy that ellminatas depend- 
ency between packets. 

26. A system comprising; 

means for encoding streaming video data; 

means for selecting a set of encoded bitstream segments based on a contribution to distortion reduction; and 
rneans for generating data packets containing the sat of encoded bitstream segments such that the packetized 
bitstream segments are arranged to reduce dependency between individual ones of the data packets. 

27. The system as recited in claim 26 wherein the generating means further comprises; 

means for organizing bitstream segments within a data packet based on contributions to distortion reduction 
associated with individual ones of the bitstream segments. 

28. The system as recited in claim 26 wherein the generating means further comprises; 

means for organizing bitstream segments within a data packet In a decreasing order based on contributions 
to distortion reduction associated with individual ones of the bitstream segments. 

29. The system as recited In claim 26 further comprising; 



means for estimating a contribution to distortion reduction associated with individual ones of the encoded 
bitstream segments. 

30. The system as recited In claim 26 further comprising; 

means for selecting the set of encoded bitstream segments based on an available bandwidth. 

31. One or more computer-readable media comprising computer executable instructions that, whan executed direct 
a computing system to; 

calculate a contribution to distortion reduction associated with each of a plurality of encoded video bitstream 
segments; 

determine a subset of the plurality of encoded video bitstream segments based on a comparison of the con- 
tribution to distortion reduction; and 

generate a data packet containing the selected subset of video bitstream segments. 

32. The one or more computer-readable media as recited in claim 31 further comprising computer executable Instruc- 
tions that, when executed, direct a computing system to; 

calculate the contribution to distortion reduction according to a formula: 

ZiD(W)^ADo(fM^^D^W) 



wherein; 
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® associated with an /«- frame, an fi> bitplane. 

® ‘^'^'’® aaPPression of the bitstream segment associated with the /«• frame. bitplane. 

computer-readable media as recited in deim 32 further comprising computer executable Instruc- 
tions that, when executed, direct a computing system to: 

calculate the drifting suppression according to a foimula: 

= ADo(/ ,/,0 X Z[n^"Cr+ -h *) X Rhl{J ^ n)j 

Wherein: 

® bitstream se^nent associatad with an <«• frame, an /»« bitplane. 

«h>/ 9 represents a percentage of hIgh-predIctlon-high-refofBnce (HPHR) modes in an P* frame- 
Phl(0 iB^ente a percentage of high-predlction-lovir-reference (HPLR) modes In an P» frame- md 
/Visa number of frames used for drifting estimation, »- irwne. ana 
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Macro Blocks 
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